Everyone’s Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

In NLP annotation, it is common to have multiple annotators label the text and then obtain ground truth labels based on major annotators’ agreement. However, are individuals with different backgrounds various voices. When annotation tasks become subjective, such as detecting politeness, offense, social norms, voices differ vary. Their diverse may represent true distribution of people’s opinions subjective matters. Therefore, crucial study disagreement from understand which content controversial annotators. our research, we extract five datasets, fine-tune language models predict disagreement. Our results show that knowing demographic information (e.g., gender, ethnicity, education level), in addition task text, helps To investigate effect demographics their level, simulate combinations artificial explore variance prediction distinguish inherent controversy perspective. Overall, propose an innovative mechanism for better design process will achieve more accurate inclusive systems. code dataset publicly available.